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progress than comparable students who attended other district schools. The 
paper provides a longitudinal examination of the students ' academic 
achievement during the school's first 3 years of operation, 1996-1997 through 
1998-1999. The study also extends the range of statistical techniques 
commonly used in program evaluations. Hierarchical linear modeling (HLM) was 
used in an individual growth curve analysis to examine the students' 
achievement in reading and mathematics. Project and control groups' reading 
and mathematics scores on the Stanford Achievement Test were compared. Two 
panels of students in the project school were compared. Panel A was comprised 
of 114 students initially enrolled in grade 2, and Panel B of 159 students 
initially enrolled in grade 3. Using stratified random sampling, equal 
numbers of students who attended other district schools were selected as 
control groups. Analyses were completed in reading and mathematics. Results 
of the HLM analyses indicate that significant levels of growth were achieved 
across the 3-year period in both subject areas for all groups of students. In 
reading, there were no statistically significant differences between project 
and control groups that could be attributed to group membership, but in 
mathematics the results were less consistent. In Panel A, no significant 
source of between-group variation was identified. In Panel B, however, the 
rate of growth in mathematics over time was greater for project students than 
for control students. This was the only statistically significant difference 
attributed to group membership across the analyses. After 3 years of 
instruction through the Edison model, reading and mathematics performance 
levels did not differ significantly from those of students in other district 
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ABSTRACT 



a 



Current initiatives in school reform include contracting educational services to private, 
for-profit firms. The purpose of this paper is to ascertain whether the students who attended one 
elementary school in Miami-Dade County, run by The Edison Project (currently Edison Schools 
Inc.), made greater academic progress than comparable students who attended other district 
schools. The paper provides a longitudinal examination of the students’ academic achievement 
during the school’s first three years of operation, 1996-97 through 1998-99. In addition, it 
extends the range of statistical techniques typically used in program evaluations. Hierarchical 
linear modeling (HLM) was employed in an individual growth curve analysis to examine the 
students’ achievement in reading and mathematics. 

The project and control groups’ reading and mathematics scores on the Stanford 
Achievement Test were compared. Two panels of students in the project school were followed 
across the three school years. Panel A was comprised of 1 14 students initially enrolled in grade 
2, and Panel B of 159 students initially enrolled in grade 3. Using stratified random sampling, 
equal numbers of students who attended other district schools were selected as control groups. 
During the span of the study, the selected control groups remained comparable to the project 
groups in terms of demographic characteristics and performance on a pretest, despite attrition. 

Analyses were completed in reading and mathematics for both Panels A and B. The 
results of the HLM analyses indicated that significant levels of growth were achieved across the 
three-year period in both subject areas for all groups of students. In reading, there were no 
statistically significant differences between the project and control groups that could be 
attributed to group membership. However, the results were less consistent in mathematics. In 
Panel A, no significant source of between-group variation was identified. However, in Panel B, 
the rate of growth in mathematics over time was greater for the project students than for the 
control students. This was the only statistically significant difference attributed to group 
membership across the four analyses. Nonetheless, after the project students had received three 
years of instruction by means of the Edison model, their final levels of performance in reading 
and mathematics did not differ significantly from the students’ in other district schools. 
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INTRODUCTION 

Fueled by widespread dissatisfaction with the state of the public school system, 
experiments in the privatization of education are on the rise. Whether through vouchers, charter 
schools, or contracts for the provision of services, private firms are having an impact on public 
education. Amidst promises of increased achievement from potential service providers and 
resounding denouncements from their opponents, these experiments are rarely subjected to 
sound, objective examination. 

Miami-Dade County Public Schools has been receptive to the idea of privatization. On 
December 13, 1995, the School Board of Miami-Dade County approved a five-year contract with 
The Edison Project (currently Edison Schools Inc.) to manage an elementary school in the 
district. Funding for the project school was comparable to other schools in the district, but 
adjusted to accommodate the unique aspects of its operation. The Edison Project was permitted 
to retain any excess funds as profit. 

The primary goal of this research was to address the following question: 

■ Did the students who attended The Edison Project’s school for three years make 
greater academic progress than comparable students who attended other schools in 
the district? 
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BACKGROUND INFORMATION 

The term “privatize” first appeared in a dictionary in 1983. It was narrowly defined as 
“to make private, especially to change (as a business or industry) from public to private control 
or ownership” (Savas, 1987, p.3). Although the term privatization is relatively new, private 
entities have been contracted to perform public services throughout history. 

Recently, under the umbrella of school reform, experiments have been carried out in 
privatization under various forms. Each form shifts the balance of power between the public and 
private sectors, and further blurs the public-private boundaries. Murphy (1996) described the 
five main forms of privatization: (a) franchises, (b) vouchers, (c) grants/subsidies, (d) 
deregulation, and (f) contracting. In a franchise, the government gives a private entity the 
authority to provide a service within a specific geographic area. The source of payment 
distinguishes a franchise: the users pay the provider directly. In the second form of privatization, 
vouchers authorize consumers to purchase specific goods or services from the private market. 
The government specifies who is eligible to purchase a service, and who is eligible to provide the 
service. Vouchers keep the financing in the public sector, while distributing the purchasing 
power to eligible consumers. As such, vouchers encourage consumption by a specific group. 

By awarding grants and subsidies, the third form of privatization, the government 
provides financial aid or contributions (cash or in-kind) to private organizations or individuals to 
encourage them to produce a service. Grants and subsidies serve to make the private sector 
responsive to needs in the public sector and to encourage them to produce a service at a reduced 
cost to the users. While grants serve to encourage the provision of social services by non-profit 
organizations, tax programs serve to encourage the donation of materials, equipment, and other 
goods by private industries. Grants are offered to organizations, colleges and universities, 
districts, schools, and teachers to institute change in public schools. 

The fourth type of privatization is deregulation. Deregulation opens an industry to 
competitive pressures by removing government regulations. It increases the number of providers 
of a service by weakening or removing public controls. Charter schools represent a form of 
school deregulation, designed to provide alternatives to traditional public schools. The Charter 
School Expansion Act of 1998 authorized states to use federal funds for planning, designing, and 
initial implementation of public charter schools (S.1380, 1998). By 1999, charter school laws 
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had been enacted in 36 states, Puerto Rico, and the District of Columbia. These laws permit 
limited numbers of schools to seek contractor status (WestEd and U.S. Department of Education, 
2000; Center for Educational Reform, 1999; Hill, Pierce, and Guthrie, 1997). Generally, charter 
schools function autonomously from the school district, with fewer regulations. Accountability 
is minimal, as long as the stipulations of the charter are carried out. 

Through the final form of privatization, contracting, governments purchase services from 
the private sector, either from for-profit or not-for-profit organizations. The government retains 
responsibility for the service, but the private firm performs the service. A whole service or 
certain elements of the service may be purchased. This form of privatization is the focus of this 
study. 

There have been only four for-profit companies recently engaged in contract-based 
privatization efforts. These are: Educational Alternatives Inc. (EAI), and Public Strategies 
Groups Inc., which are based in Minneapolis, Minnesota; Alternate Public Schools (APS), based 
in Nashville, Tennessee; and The Edison Project (currently Edison Schools Inc.), based in New 
York (Edison Schools, 2001; Brown and Hunter, 1996; Editorial Projects in Education, 1998; 
Feir, 1996; Schrag, 1996; Tetreault and Picus, 1996). 

As previously noted, the Miami-Dade County Public Schools (MDCPS) has been 
receptive to the idea of privatization. In fact, EAI’s five-year contract to operate an elementary 
school in Miami-Dade County, signed in June 1991, represents the first time that a public school 
system entered into a partnership with a private firm to provide the complete educational 
program. The students’ academic performance during the first three years of the program’s 
operation (1991-92 through 1993-94) was evaluated by the district. It was shown that the 
students’ academic performance generally improved in the project school, but not to the degree 
anticipated. Students in a control group, who did not benefit from the program provided by EAI, 
performed comparably well (Abella, 1994). Consequently, the district did not renew its contract 
with EAI. 

Currently, only The Edison Project remains as a viable contender in the privatization 
movement. The Edison Project was founded by Chris Whittle, originator of the highly profitable 
in-school news program Channel One (Schrag, 1996). A group of nationally recognized 
professionals from the fields of education, technology, government, communications, and 
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business has worked to develop and guide the implementation of the Edison model. The 
company’s first four schools were opened in 1995, in Mt. Clemens, Michigan; Wichita, Kansas; 
and Sherman, Texas. That year it also began oversight of the operation of a charter school in 
Boston, Massachusetts for a non-profit agency. The number of schools operated by Edison has 
increased yearly, by expansion and acquisition of new contracts. During the 2001-02 school 
year, The Edison Project is operating 136 public schools in 23 states, including some charter 
schools (The Edison Project, 1999; Edison Schools, 2001). 

In December 1995, the MDCPS signed a five-year contract with The Edison Project. The 
contract called for The Edison Project to manage an elementary school in the district from 
August, 1996 through June, 2001. The selected school, which was newly constructed, first 
opened under management by The Edison Project. The school drew students from an attendance 
area established by the district. The Edison Project received funding comparable to other 
schools in the district, but adjusted to accommodate the unique aspects of its operation (e.g., an 
extended school year). The Edison Project was permitted to retain any excess revenues over 
expenditures as its compensation for the services provided. 

The Edison model is not simply a new instructional method. The implementation of the 
model calls for global changes in a school’s operation. It requires fundamental changes in a 
school’s organization, schedule, curriculum, staffing, technology, as well as instructional 
method. 

The Edison Model 

The Edison model is described in the company’s book, Partnership School Design . 
According to this document, the model is founded on ten basic principles, which are known as 
“fundamentals.” While the document contends that the application of any of the fundamentals 
would result in an improvement, a strategy that encompassed all ten would achieve dramatic 
results. The fundamentals are: 

1 . Schools organized for every student’s success 

2. A better use of time 

3. A rich and challenging curriculum 

4. Teaching methods that motivate 

5. Assessment that provides accountability 
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6. Educators who are true professionals 

7. Technology for an Information Age 

8. A partnership with families 

9. Schools tailored to the community 

10. The advantages of system and scale (The Edison Project, 1994a, p. 10). 

The implications of each fundamental on the operation of an Edison project school are 
detailed in the following sections. 

Organization 

The organization of an Edison project school is quite distinct from a typical school in 
Miami-Dade County. The project school is organized into “schools-within-a-school,” which are 
known as “academies.” Within each academy, students are organized into “houses.” Teachers 
assigned to a house work as a team in teaching the core subjects of the curriculum. Each house 
includes 90 to 120 students, and includes an equal number of students from each age/grade level 
in the academy. This mixture of ages/grades facilitates the forming of ability groups for 
instruction and related activities. Flexible, ad hoc ability groups are a key element in the 
instructional strategy of the Edison model. Such grouping is distinct from tracking, which the 
model shuns. In a project school, every student receives the same curriculum. The teacher- 
student ratio of a project school is approximately 1:19, but the Edison model’s emphasis is not 
necessarily on small classes. According to Partnership School Design , “the key to effective class 
organization is not the staffing ratio or the average class size, but matching the class and staffing 
structure to instructional purpose” (The Edison Project, 1994a, p. 19). To facilitate this, Edison 
model emphasizes flexible scheduling. 

Time 

Perhaps the most critical element in an educational program is the “time on task;” this 
represents the time actually devoted to instruction. Yet, the United States has one of the shortest 
school years of the industrialized world. To remedy this, the Edison model is based on an 
extended school year. In a project school, the school year typically begins August 15 and ends 
July 1. It consists of 210 school days, which is the same as schools in Japan. In contrast, the 
school year in Miami-Dade County consists of only 180 days. 
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The school day is also longer in an Edison project school than in the typical Miami-Dade 
County school. Depending on the grade level, the school day in a project school is from 50 to 90 
minutes longer. Such differences may seem minor; but when combined with the extended school 
year, they produce a remarkable increase in the amount of time a student spends in school during 
the 13 years between kindergarten and grade 12. A student who remains in project schools for 
the duration will have a total of 21,210 hours in school. In contrast, a student who attends the 
typical Miami-Dade County schools will have only 15,570 hours. The difference, when adjusted 
to a 180-day school year with a seven-hour day, represents almost 4 J A years of additional 
schooling (Gomez and Shay, 1998, 1999, 2000). This is a sizable increase in the time on task. 
Curriculum 

The Edison curriculum is holistic in nature. The curriculum is divided not so much into 
subjects but domains. There are five major domains: (a) humanities and the arts, (b) 

mathematics and science, (c) character and ethics, (d) health and physical fitness, and (e) 
practical arts and skills. The domains serve to integrate the curriculum, so teachers can employ 
an interdisciplinary approach in the delivery of instruction. Lessons are organized around 
projects or real life problems, which require the students to delve into different disciplines. 

The Edison curriculum is also results-oriented. Each academy has a set of over 100 
explicit standards that define the level of educational development that students are expected to 
achieve (The Edison Project, 1994b, 1994c). The students must demonstrate the attainment of 
these standards before they can be promoted to the next academy. The standards establish very 
high academic expectations for the students. 

Teaching Methods 

The developers of the Edison instructional program recognize the fact that students leam 
in different ways, so the program employs a variety of teaching methods. They range from 
traditional lectures to such innovative methods as cooperative learning. However, all the 
methods, according to The Edison Project, have one thing in common: their effectiveness is well 
documented. 

Assessment 

A unique assessment system was developed for the Edison model. It was designed to 
correspond with the objectives of the educational program. The components of the system are: 
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the Quarterly Learning Contract, the student portfolio, the embedded assessments, and the 
Academy Promotion Review. The system is used to monitor the progress of the students in 
attaining the standards of the educational program. Beyond the assessment measures of the 
Edison model, students in a project school also take all the standardized tests required by the 
district and the state. The results of these tests enable the project school to gauge the academic 
progress of the students against local and national norms. 

Educators 

The best teachers are often promoted right out of the classroom. To curtail this loss, the 
Edison model employs a career ladder for teachers. Professional development is an integral part 
of the ladder. The professional development program is carefully aligned with the model’s 
elements of curriculum, instruction and assessment. Closely associated with professional 
development is the assessment of job performance. The job performance of the teachers in a 
project school is assessed by a results-oriented system that includes: (a) the students’ academic 
progress, (b) the quality of the curriculum units developed, (c) the fluency with various 
instructional methods, and (d) the relationship with colleagues, students, and their families. 
Evidence of the teacher’s accomplishments in these areas is maintained in a portfolio. 

Technology 

The use of sophisticated technology in the classroom is not a new idea. However, an 
Edison Project school is unique in the extent to which it uses such technology. A computer is 
made available to every student’s family, every teacher and the principal; all these computers are 
linked by a network. In this manner, communication among these parties is facilitated. 
Partnership 

A project school attempts to engage the students’ families in the educational process. 
Communication is maintained through quarterly parent meetings, by telephone and through the 
computer network. Additionally, if the space is available, a project school maintains a Parent 
Center, which serves as a base for family members who visit the school. Finally, a Parent 
Advisory Council is convened by the principal which provides a forum for family members to 
express their opinions and ideas regarding the school’s programs and policies. 
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Community 

An Edison project school is tailored to the community it serves. About one-fourth of the 
school’s curriculum is determined locally. This ensures that the educational program is in accord 
with the priorities and concerns of the community. Furthermore, the project school is designed 
to be a hub of activity for the community. The project school enlists the cooperation of 
community agencies and professionals in the delivery of social services to meet the specific 
needs of the students in the project school. 

System and Scale 

Project schools are linked both in their common purpose and literally through the 
computer network. The network facilitates the exchange of materials and ideas for improving 
the educational program. In addition, the Central Services of The Edison Project provides 
support, guidance, and resources to the project schools. 

Overall, the ten “fundamentals” which comprise the Edison model make up an eclectic 
mixture of elements that are not necessarily new or unusual. They are based on established 
educational practices and sound research. However, the merger of these elements into a cohesive 
strategy is unique to the Edison model. The basic elements of the model are summarized in 
Table 1 . 




Table 1 





Basic Elements of the Edison Model 


Fundamentals a 


Elements 


1. Organization 


“Schools-within-a-school” consisting of academies that 
combine several ages/grade levels 

Ability grouping, but no tracking 

Teachers organized into teams 

Flexible scheduling (e.g., block scheduling) 


2. Time 


Extended school year 
Extended school day 


3. Curriculum 


Interdisciplinary curriculum 

Lessons organized around projects or real life problems 

Results-oriented standards for promotion to the next 
academy 


4. Teaching Methods 

5. Assessment 


Variety of teaching methods 

Unique assessment system for monitoring students’ 
progress toward standards 


6. Educators 


Career ladder for teachers 

Emphasis on professional development 

Results-oriented system for assessing job performance 
of teachers 


7. Technology 


Emphasis on technology (e.g., computer provided for 
each student’s home) 

Home and school are linked by computer network 


8. Partnership 

9. Community 


Emphasis on parental involvement 
One-fourth of the curriculum determined locally 
School is conduit for social services 


10. System and Scale 


School is supported by Central Services of The Edison 
Project 

School and Central Services are linked by computer 
network 



a For the full text of the fundamentals, see pages 5-6. 
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This study was designed to examine the academic achievement of the students in the 
Miami-Dade County Public Schools (MDCPS) who attended the elementary school operated by 
The Edison Project (currently Edison Schools Inc.). To gauge the students’ academic 
performance, a quasi-experiment was conducted. A quasi-experiment is a technically acceptable 
alternative to a true experiment. Quasi-experiments are often used in “natural social settings” 
where a true experiment is not feasible (Campbell and Stanley, 1963). The specific quasi- 
experimental design used in this study was a form of the “nonequivalent control group design.” 
This design essentially involved using repeated test scores to compare the performance of the 
students who were exposed to an experimental treatment (i.e., the experimental group) to that of 
a group who were not (i.e., the control group). The two groups were considered 
“nonequivalent,” because the subjects were not randomly distributed between them (as is the 
case with a true experimental design). This design controlled for most of the primary threats to 
the internal validity of the findings, including history, maturation, testing, instrumentation, and 
mortality. 

In applying the nonequivalent control group design to the assessment of the project 
students’ academic achievement, the Edison model represented the experimental treatment. As 
such, the students who attended the project school comprised the experimental group. The 
students in the control group were drawn from a pool of all 1996-97 MDCPS students who did 
not attend the project school. The Stanford Achievement Tests (SAT) administered during the 
students’ passage through grade levels of the project school were used to measure their academic 
performance. The initial test in the series (i.e., the pretest) was administered in the 1995-96 
school year. It was the last SAT test administered prior to the students’ entry into the project 
school in year 1 (1996-97). 

The analysis was limited to the students who enrolled in the project school in year 1 and 
attended for three years. These two constraints ensured that the students included in the analyses 
had the greatest exposure to the Edison model. The analysis focused on two panels of students: 
those in grade 2, designated Panel A; and those in grade 3, designated Panel B. Each panel 
represents a single grade-level group that was followed as it progressed through the three years 
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of the study. An overview of the subjects’ passage through the grade levels of the project school 
is presented in Table 2. 

Table 2 



Grade Level Progression of the Panels through the Project School 







Grade Level 






Year 1 
1996-97 


Year 2 
1997-98 


Year 3 
1998-99 


Panel A: 


2 


3 


4 


Panel B: 


3 


4 


5 


Note. While the project spans five school years from 1996-97 through 1999-2000, the 
research is limited to the first three years. 



Sampling Procedures 

As stated previously, all students in grades 2 and 3 who were enrolled in the project 
school during year 1 (1996-97) were part of the experimental treatment group. In that year, a 
total of 1077 students were enrolled in the project school. This included 181 enrolled in grade 2 
and 193 enrolled in grade 3. 

The students in the control group were drawn from a pool of 1996-97 MDCPS students. 
As such, the sampling frame for the control group consisted of all second and third grade 
students in the district who did not attend the project school. The pool of potential control group 
members for Panel A contained 27,743 students; and the pool for Panel B contained 27,275 
students. 

Stratified random sampling was used in the selection of the control group to ensure that it 
corresponded proportionally with the experimental group on the following variables: grade level, 
ethnicity (i.e., black, non-Hispanic; Hispanic; white, non-Hispanic; other), eligibility for the 
free/reduced lunch program (i.e., eligible; not eligible), and performance on the pretest (i.e., 
reading and mathematics percentile scores, grouped into ten-percentile-point bands). Selection 
of the sample entailed the following steps. First, a seven character “group code” was assigned to 
each second and third grade student in the project school. A student’s group code corresponded 
to their characteristic on each of the above variables. As such, 2HY2040 could be the group 
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code for a second grade (2), Hispanic (H) student who is eligible for the ffee/reduced lunch 
program (Y), and scored at the 23 rd percentile on the reading pretest (20) and at the 46 th 
percentile on the mathematics pretest (40). Each potential control group student was also 
assigned a group code, along with a randomly generated “pool number.” Within group codes, 
the control group students were then sorted in numerical order, based on their pool numbers. 
The control group was then selected from the ordered pool, in proportionally equal numbers to 
the project group, by group code. This procedure provided a stratified random sample, 
comparable to the project students based on grade level, ethnicity, socioeconomic status, and 
initial ability level. 

As was the case for the students in the project school, the selected sample of control 
group students was also followed longitudinally across the three school years, 1996-97 through 
1998-99. Because the students in the project school were all enrolled in the same school for the 
duration of the study, students in the control group were removed from the study if they changed 
schools during the three-year period. This controlled for the potentially confounding effects of 
discontinuity in educational programs. 

Composition of the Sample 

Many of the project students lacked pretest scores from the 1995-96 school year, prior to 
enrollment in the project school. Pretest scores may have been missing for a student for several 
reasons, for example: absence on the day of testing, recent transfer into the school district, or 
enrollment in the exceptional student education or the English for Speakers of Other Languages 
programs. Students without at least one pretest score (i.e., reading or mathematics) were not 
eligible for inclusion in the quasi-experiment. Thus, of the students originally enrolled in the 
targeted grades, only 1 14 of the 181 enrolled Panel A students; and 159 of the 193 enrolled Panel 
B students were included in the study. Corresponding numbers of students were consequently 
selected to comprise the control groups. 

Table 3 shows the number of students in each group through the three years of the 
project. The numbers given represent students who were included in at least one statistical 
procedure completed for this study. Overall, 74.91% of the students identified for inclusion in 
the study remained through the end of year 3, resulting in an overall attrition rate of 25.09%. 
Somewhat higher percentages of the project students were lost due to attrition than their 
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counterparts in the control group. This was particularly true in Panel B, where the difference in 
rate of attrition was more than ten percent. 

Table 3 

Number of Students in the Project and Control Groups by Year 



Panel Members Attrition 

Year 1 Year 2 Year 3 Rate 



Panel A 


Project 


114 


95 


83 


27.19% 




Control 


114 


108 


91 


20.18% 


Panel B 


Project 


159 


130 


108 


32.08 % 




Control 


159 


153 


127 


20.13 % 



Nonetheless, the groups remained similar in terms of demographic characteristics, despite 
attrition. A delineation of the demographic characteristics of each group may be found in Table 
4. As the table shows, the groups were comprised primarily of black students eligible for the 
free/ reduced lunch program. Few had been enrolled in the English for speakers of other 
languages (ESOL) program or the exceptional student education (ESE) program. 

Initial Comparability of the Groups 

The effect of the differential levels of attrition on the comparability of the groups in terms 
of their initial ability levels was also examined. Simple Analyses of Variance (ANOVAs) were 
performed at the end of the three-year period, comparing each panel’s pretest scores in reading 
and mathematics. Students were only included in the analyses if they remained in the group over 
the three years of the study. The SAS GLM procedure was used to accommodate the unequal 
number of students in the groups (SAS Institute, 1985). The pretest scores were the students’ 
Total Reading and Total Mathematics scores on the Stanford Achievement Test (SAT) 
administered during the 1995-96 school year, prior to the project students’ enrollment in the 
project school. Table 5 shows the groups’ mean pretest scores. 
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Table 4 

Demographic Characteristics of the Project and Control Groups 



14 







Panel A 


Panel B 


enaractenstic 


Project 
% (n) 


Control 

% (n) 


Project 
% (n) 


Control 
% (n) 


Gender 


Female 

Male 


53.01(44) 

46.99(39) 


52.75(48) 

47.25(43) 


49.07(53) 

50.93(55) 


51.18(65) 

48.82(62) 


Ethnicity 


Black 

Hispanic 

White 


90.36(75) 

8.43(7) 

1.20(1) 


86.81(79) 

12.09(11) 

1.10(1) 


87.96(95) 
12.04(13) 
-- (0) 


81.89(104) 
18.11(23) 
-- (0) 


F/R Lunch 


Eligible 
Not Eligible 


90.36(75) 

9.64(8) 


87.91(80) 

12.09(11) 


91.67(99) 

8.33(9) 


86.61(110) 

13.39(17) 


ESOL 


Enrolled/Former 
Not Enrolled 


21.69(18) 

78.31(65) 


15.38(14) 

84.62(77) 


34.26(37) 

65.74(71) 


30.71(39) 

69.29(88) 


ESE 


ESE 

Gifted 

Not Enrolled 


1.20(1) 

16.87(14) 

81.93(68) 


1.10(1) 

8.79(8) 

90.11(82) 


2.78(3) 

6.48(7) 

90.74(98) 


9.45(12) 

4.72(6) 

85.83(109) 



Note . F/R Lunch = Free/Reduced Lunch Program; ESOL = English for Speakers of Other Languages 



program; ESE = Exceptional Student Education Program 



The ANOVAs revealed that there were no significant differences in the pretest scores of 
the project and control groups. Hence, it may be concluded that the initial ability levels of the 
groups remained comparable despite attrition. Subsequent differences in reading and 
mathematics achievement may then be attributed to the intervening instructional experiences. 

Table 5 



Mean Pretest Scores of the Project and Control Groups 





Mean 


READING 

Standard 

Deviation 


n 


MATHEMATICS 
Standard 
Mean Deviation 


n 


Panel A 














Project 


515.73 


38.75 


77 


531.33 


41.71 


80 


Control 


512.33 


33.26 


84 


524.86 


39.21 


87 


Panel B 














Project 


545.84 


33.40 


98 


550.84 


41.24 


104 


Control 


545.24 


31.20 


119 


549.38 


39.38 


118 



Note . The ANOVA analyses indicated that there were no significant differences in the groups’ mean 
pretest scores at the end of the three-year period. 
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Instrumentation 

The students’ performance on the Stanford Achievement Test, Eighth Edition (SAT) was 
the primary source of data. Scaled scores were used in all analyses because they are articulated 
across grade levels. As previously noted, the pretest scores, used as a measurement of initial 
status in the selection of the comparison group, were the Total Reading and Total Mathematics 
scores. These scores were composite scores of all subtests administered in the appropriate 
content domain. They were selected because preliminary analyses indicated that they were the 
best predictors of future achievement. Beginning in the 1997-98 school year, however, the 
district elected to reduce the number of subtests administered. Only the Reading Comprehension 
and Mathematics Applications subtests were administered in subsequent years. As a result, the 
Reading Comprehension and Mathematics Applications subtest scores served as the dependent 
variables in this study. 

Data Analysis Strategies 

Hierarchical linear modeling (HLM) was used in a growth curve analysis of the quasi- 
experiment. Four sets of analyses were completed: Panel A in reading and mathematics, and 
Panel B in reading and mathematics. The use of HLM in the study of academic growth has been 
well documented, although it has not typically been used in the evaluation of educational 
programs (Burchinal, Baily, & Snyder, 1994; Rogosa & Saner, 1995; Willet, Singer, & Martin, 
1998; Williamson 1990; Williamson, Applebaum, & Epanchin, 1991; Woodruff & Houston, 
1992). The HLM procedure provides an advantage over traditional regression techniques 
because it partitions variance into embedded hierarchical structures inherent in the nature of the 
data. In addition, rather than focusing only on mean values at discrete points in time, like 
traditional methods, HLM takes into account the rate of growth at each structural level. In the 
case of growth curve analyses, the model explicitly represents individual growth. In addition, it 
is possible to examine differences in the students’ achievement levels and rates of growth within 
and between groups (e.g., schools, classrooms, or experimental treatment groups). 

There are several other advantages to HLM. First, it is more flexible in its requirements 
for the data. The number of observations available for each subject may vary; thus missing 
values do not pose a problem. The spacing of observations may also vary, accommodating 
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different ages or testing times across subjects. Complex models can also be specified that 
include other predictors to account for further systematic variation. Moreover, specification of 
the covariance structure is flexible. It is possible to conduct hypothesis tests about the 
determinants of the structure. 

HLM does not have the restrictive data requirements and assumptions of traditional 
multiple regression techniques. However, when the conditions and assumptions of traditional 
techniques are met, the results of HLM analyses can be formally related to those of traditional 
multiple regression. Perhaps the most compelling advantage of HLM is that it allows for the 
examination of embedded relationships in naturally occurring groups. This leads naturally to 
the study of the relationships between organizational structures and growth. 

Bryk, Raudenbush, and Congdon’s computer program HLM, Version 4, was used to fit 
the model separately for each panel and subject area (1996). In a three-level model for time- 
ordered data, the Level 1 model is comprised of individual growth trajectories. The Level 2 
model represents the variation in growth parameters among students in the treatment groups. 
And, the Level 3 model describes the variation between the two treatments (Bryk & Raudenbush, 
1992). 

The Level 1 individual growth model assumes that the students’ rate of growth is 
constant between each test period. It examines the rate of growth over time for students across 
all treatment groups. Empirical Bayes estimates are used to construct growth curves based on 
estimated true scores for individual students. Empirical Bayes estimates provide the best 
estimates of true growth. When only three measures are available , the model only predicts 
linear growth. The Level 1 model may be shown as: 

Y,ij = Kay + + e tiJ 

where: Y tij is the outcome at time t for child i in group j\ 

(Time) tjj is the time of testing; 

n 0ij is the status of child ij, that is the expected outcome for that child at the intercept 
(i.e., when Time=0); 

7 T Uj is the learning rate for child ij during the period of the study; and, 
e tij is the person-specific variance. 
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The focus of this analysis was on differences in achievement between the project and 
control groups at the end of the three-year period. As such, the variable ‘Time’ was coded so 
that the intercept reflected final status (i.e., level of achievement at the end of year 3). To 
evaluate final achievement levels, the intercept had to be centered at year 3. Thus, the time 
variable was coded as -2 for year 1,-1 for year 2, and 0 for year 3. 

At Level 2, the model examines variance in the students’ achievement within the 
treatment groups. The model at Level 2 is 

^Oij = Poo j r Oij 
n Uj = PlOj + V 

And, at Level 3 the model examines variance in the students’ achievement that may be 
attributed to group membership. It may be shown as 

Poo j ~ y 000 M 00 j 

P\oj ~ Y ioo ^ u \o ]• 

At Levels 2 and 3, J3 00j represented the mean status at the end of year 3, within 
treatment group j, while y 000 was the overall final mean status. Similarly, J3 l0j represented the 
mean growth rate within treatment group j, while y l00 was the overall mean growth rate. 
Random effects at Level 2 were represented as r oij and r Uj ; while u 00j and u ]0J represented 

random group effects. This analysis allowed the rates of growth (slopes) over the three-year 
period of the study to be examined for the individual students, in addition to the rates of growth 
within and between treatment groups. 
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The hierarchical linear modeling (HLM) procedure provides an advantage over 
traditional regression techniques, since it permits the partitioning of variance into embedded 
hierarchical structures inherent in the nature of the data. In this study, three-level HLM models 
were used to examine variation of individual students’ performance over time; variation among 
students within the same group; and variation attributable to group membership. In addition, 
rather than focussing only on mean values at discrete points in time like traditional regression, 
HLM takes into account the rate of growth, or slope, at each level of the model. 

Bryk, Raudenbush, and Congdon’s computer program HLM, Version 4 was used to 
construct the three-level models employed (1996). Four analyses were carried out: Panel A in 
reading and mathematics, and Panel B in reading and mathematics. Specifically, the HLM 
analyses were based on the students’ scaled scores on the Reading Comprehension and 
Mathematics Applications subtests of the SAT, administered in the 1996-97, 1997-98, and 1998- 
99 school years (i.e., years 1, 2, and 3). 

Panel A in Reading 

A synopsis of the results of the HLM analyses for Panel A in reading is given in Table 6. 
The table is divided into three sections. The top section notes the fixed effect of the model (i.e., 
within student variation); and the center section notes the random effects (i.e., within and 
between group variation). The bottom section lists the percentage of the overall variance that 
was accounted for by between-groups variance in the status and growth rates. It may be recalled 
that the time variable was coded so that ‘status’ represents the final level of achievement at year 3. 

Table 6 shows that the mean estimated true status of the Panel A students in reading was 
61 1.80 at the end of the third year. The mean growth rate was 23.89 scaled score points per year. 
In other words, Panel A students’ reading scores increased significantly from the initial level as 
they progressed from second through fourth grade. 

Proceeding to the middle section of the table, it may be seen that there was significant 
variability among students in the same group (i.e., Level 2) in terms of their status, or level of 
achievement at year 3 (% 2 =687.67, df=l 71, p<000), but not in terms of their rate of growth 
(^ 2 =161.27, df= 171, p>.500). Thus, while students’ achievement levels varied significantly 
within the groups, their rate of growth did not. Finally, the analysis of the group level 
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parameters (i.e., Level 3) indicated that there was no significant variability that could be 
attributed to group membership, either in terms of their status at year 3 or in their rate of growth. 

Table 6 

Three Level HLM Analysis for Panel A in Reading 



FIXED EFFECT 


Coefficient 


Standard 

Error 


t -ratio 


p value 


Mean status at year 3, /ooo 


611.80 


2.85 


215.00 


.000 


Mean growth rate, ^100 


23.89 


1.10 


21.69 


.000 


RANDOM EFFECT 


Variance 

Component 


df 


Z 2 


p value 


Level 1 










Temporal variation, e tij 


404.83 








Level 2 (students within groups) 










Individual status at year 3, r 0ij 


1054.79 


171 


687.67 


.000 


Individual growth rate, r Uj 


1.44 


171 


161.27 


>.500 


Level 3 (between groups) 










Group mean status, u 0 oj 


0.01 


1 


0.69 


>.500 


Group mean growth, uj 0j 


0.00 


1 


0.33 


>.500 


LEVEL 1 COEFFICIENT 


Percentage of variance between groups 






Status at year 3, K Qij 


.000 








Growth rate, K Vl - 


.000 









The bottom section of the table displays the percentage of overall variance that was 
attributable to between-groups variance for each parameter at Level 1 of the model. As in Bryk 
and Raudenbush (1992), this ratio was calculated as: group mean variance / (group mean 

variance + individual variance). For example, this ratio would be as follows for reading in Panel 
A: .01 / (1054.79 + .01) = .00001. As shown at the bottom of Table 6, the variance ratios for 
both status and growth indicate that the percentage of the overall variance due to between-groups 
variance was negligible (both ratios = .000). 

To aid in the interpretation of these results, Figure 1 graphically illustrates the estimated 
growth curves in reading for 30 randomly selected students in each group of Panel A. The line 
graph on the left depicts the performance of the students in the project group, while the one on 
the right depicts that of the students in the control group. It may be recalled that growth curves 
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represent linear growth in models utilizing three observations per students. The growth curves 
are based on individual student parameters computed by the HLM program using an empirical 
Bayes (EB) estimation. 




Figure 1. Estimated true reading scaled scores for 30 randomly selected students in the 
project and control groups of Panel A. The predicted group means are also given, and 
represented by thick, dashed lines. 



A listing of the estimates for Panel A’s reading scores is provided in Table 7. While a 
group’s fixed values remain constant, the calculated EB estimates varied for each individual 
student. The table presents the descriptive statistics for these individual estimates. 



Table 7 



Values of Panel A’s Estimated Parameters in Reading 



Group 



n 



Group Fixed Value Individual Empirical Bayes Estimates 

Intercept Slope Intercept Slope 

Mean Standard Mean Standard 

Deviation Deviation 



Panel A 



Project 


83 


611.80 


23.88 


-1.60 


31.44 


-.02 


.24 


Control 


91 


611.80 


23.89 


1.47 


29.81 


.01 


.23 



Note. The fixed value parameters remain constant within the group. 
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The students’ estimated true scores for each year were calculated from the fixed values of 
the group’s expected parameters and their individual EB parameter estimates. As such, an 
individual’s estimated true scores at years 3, 2, and 1 were calculated as: 

Year 3 = fixed value group intercept + EB individual intercept estimate 
Year 2 = Year 3 - (fixed value group slope + EB individual slope estimate) 

Year 1 = Year 2 - (fixed value group slope + EB individual slope estimate). 

Note that the mean estimated growth curves for each group are depicted as thick, dashed 
lines on the graphs. Estimations of gradual, steady growth in reading were found for the Panel A 
students. It is evident that while the students’ level of achievement (status) differs within the 
groups, the estimated rate of growth (slope) was uniform. Furthermore, there was little 
difference apparent in the group means shown in the two illustrations. 

Panel B in Reading 

A synopsis of the results of the HLM analysis for Panel B students in reading is shown in 

Table 8. The format of the table is identical to that of Table 6. The Panel B students’ mean 

reading scaled score at the end of year 3 was 616.67. A significant positive rate of growth was 
identified, at 20.16 scaled score points per year. In other words, the Panel B students’ reading 
achievement increased significantly as they moved from third through fifth grades. Proceeding to 
the random effects of the model, significant variability among students in the same group was 
again identified in terms of the students’ achievement levels at year 3 (^ 2 =1610.72, df=231, 
p<.000), but not in terms of growth rates (^ 2 =247.51, df=231, p=.217). Thus, like their 
counterparts in Panel A, the Panel B students’ levels of achievement varied within the groups, 
but not their rates of growth. Also in congruence with Panel A, the analysis of Panel B students’ 
reading scores revealed no significant differences between groups, either in terms of status at 
year 3 or rate of growth. In fact, examinations of variance ratios for both status and growth 
indicate that the percentage of the overall variance due to between-groups variance was 
negligible (both ratios = .000). These results are shown in the bottom section of Table 8. 

To aid in the interpretation of these results, Figure 2 illustrates the estimated true growth 
curves of 30 randomly selected students in each group. The growth curves were calculated as 
described for the Panel A students. See Table 9 for a listing of the estimated parameters. A 
pattern of gradual, steady growth in reading is apparent for the students in Panel B. In addition, 
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differences in achievement levels are evident, while the rate of growth appears to be relatively 
consistent within the groups. Finally, a comparison of the mean growth curves for the two 
groups indicates that the average levels of achievement and growth appear to be similar, as 
indicated in the HLM analysis. 



Table 8 

Three Level HLM Analysis for Panel B in Reading 



FIXED EFFECT 


Coefficient 


Standard 

Error 


t ~ ratio 


p value 


Mean status at year 3, y ooo 


616.67 


2.39 


258.41 


.000 


Mean growth rate, y 100 


20.16 


0.72 


27.87 


.000 


RANDOM EFFECT 


Variance 

Component 


df 


x 1 


p value 


Level 1 










Temporal variation, e (ij 


221.08 








Level 2 (students within groups) 










Individual status at year 3, r 0ij 


1136.59 


231 


1610.72 


.000 


Individual growth rate, r Uj 


6.54 


231 


247.51 


.217 


Level 3 (between groups) 










Group mean status, u 00 j 


0.04 


1 


0.53 


>.500 


Group mean growth, u }0 j 


.00 


1 


0.27 


>.500 


LEVEL I COEFFICIENT 


Percentage of variance between groups 






Status at year 3, 7T oi j 


.000 








Growth rate, 7t Vl - 


.000 









Thus, the HLM analyses of reading scores for Panels A and B were quite consistent. As 
expected, significant growth was observed as students progressed from grade to grade. And, 
while individual students’ achievement levels varied significantly within their groups, their rates 
of growth did not. In addition, the analyses failed to reveal any significant differences in reading 
achievement levels or rates of growth that could be attributed to membership in either the project 
or control group. 
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Figure 2. Estimated true reading scaled scores for 30 randomly selected students in the 
project and control groups of Panel B. The predicted group means are also given, and 
represented by thick, dashed lines. 



Table 9 



Values of Panel B’s Estimated Parameters in Reading 



Group 



n 



Group Fixed Value Individual Empirical Bayes Estimates 

Intercept Slope Intercept Slope 

Mean Standard Mean Standard 

Deviation Deviation 



Panel B 



Project 


108 


616.65 


20.16 


-2.18 


32.51 


-.08 


1.49 


Control 


127 


616.68 


20.15 


1.86 


32.57 


.07 


1.54 



Note. The fixed value parameters remain constant within the group. 



Panel A in Mathematics 

Analyses were conducted of the students’ mathematics achievement as well. The 
analysis for Panel A is summarized in Table 10. The format of the table is identical to that of 
Tables 6 and 8. The mean scaled score for Panel A students was 620.63 at the end of the third 
year of the project. The average growth rate was found to be significantly positive at 29.34 
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scaled score points per year, indicating that the students’ scores increased as they progressed 
from second through fourth grades. 

Proceeding to the random effects of the model, significant variability was found among 
the Panel A students within the same group, both in terms of their status at year 3 ( j 2 =1241.55, 
df=171, p< 000) and in terms of their rate of growth (j 2 =291.68, df=171, p<.000). However, no 
significant source of variability was found between groups, either in terms of status or growth 
rate. In fact, the examination of variance ratios for both Level 1 parameters indicated that the 
percentage of the overall variance in mathematics scores due to between-groups variance was 
again negligible. As shown in the bottom section of Table 10, the ratios for status and growth 
were both .002. 

Table 10 



Three Level HLM Analysis for Panel A in Mathematics 



FIXED EFFECT 


Coefficient 


Standard 

Error 


t -ratio 


p value 


Mean status at year 3, fooo 


620.63 


4.36 


142.31 


.000 


Mean growth rate, Y 100 


29.34 


1.58 


18.56 


.000 


RANDOM EFFECT 


Variance 

Component 


df 


Z 2 


p value 


Level 1 










Temporal variation, e tij 


460.72 








Level 2 (students within groups) 










Individual status at year 3, r 0ij 


2470.90 


171 


1241.55 


.000 


Individual growth rate, r Uj 


166.14 


171 


291.68 


.000 


Level 3 (between groups) 










Group mean status, Uqqj 


4.90 


1 


0.74 


>.500 


Group mean growth, u }0j 


0.30 


1 


0.33 


>.500 


LEVEL 1 COEFFICIENT 


Percentage of variance between groups 






Status at year 3, 7T 0j j 


.002 








Growth rate, 7T Uj - 


.002 









Figure 3 illustrates the results of this analysis. Individual students’ estimated true scores 
in mathematics were calculated in the same way described for the reading scores. See Table 1 1 
for a listing of the estimated parameters for the groups. Again, it may be seen that the estimated 
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growth curves for 30 randomly selected students in each group showed progress over time. 
However, the differences in the rates of growth for individual students are evident in these line 
graphs. These differences are represented in the graph by the divergence and convergence of 
individual students’ growth curves. However, once again, the group means and rates of growth 
were comparable. 




Figure 3. Estimated true mathematics scaled scores for 30 randomly selected students in the 
project and control groups of Panel A. The predicted group means are also given, and 
represented by thick, dashed lines. 



Table 1 1 

Values of Panel A’s Estimated Parameters in Mathematics 



Group 



Panel A 



n 



Group Fixed Value Individual Empirical Bayes Estimates 

Intercept Slope Intercept Slope 

Mean Standard Mean Standard 

Deviation Deviation 



Project 


83 


619.27 


29.67 


-2.66 


52.66 


-.17 


10.89 


Control 


91 


621.99 


29.00 


2.43 


41.09 


.16 


1.85 



Note. The fixed value parameters remain constant within the group. 
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Panel B in Mathematics 

A synopsis of the results of the HLM analysis for Panel B students in mathematics is 
shown in Table 12. The format of the table is identical to that of Tables 6, 8 and 10. As the table 
shows, the overall mean scaled score in mathematics was 629.24 in year 3. The average growth 
rate was significantly positive at 29.87 scaled score points per year. As such, the Panel B 
students’ achievement increased significantly as they progressed from third through fifth grades. 
An examination of the random effects reveals that there was significant variation among students 
within the same group with regard to their status at the end of year 3 (^ 2 =1 172.76, df=231, 
p<.000), but not with regard to their rate of change (^ 2 =230.21, df=231, p>.500). However, at 
the third level of the model, significant variability was identified between the groups’ estimated 
mean rates of change (^ 2 =5.33, df=l, p=.020). Still, the mean differences between groups at 
year 3 were not statistically significant. 

In keeping with these findings, an examination of the variance ratios for status indicated 
that the percentage of the overall variance due to between-groups variance was negligible (.001). 
However, this was not the case for the variance ratio for growth rate. As shown at the bottom of 
Table 12, the ratio for growth rate was .528. In other words, 52.8% of the overall variance in the 
students’ growth rates was attributed to between-groups variance. 

Figure 4 provides a graphic illustration of the growth curves for Panel B students in 
mathematics. Again, the two line graphs show the estimated growth curves for 30 representative 
students in each group, with estimated true scores calculated as described previously. See Table 
13 for a listing of the estimated parameters for the groups. Also shown on each line graph, as a 
thick dashed line, is the groups’ estimated mean growth curve. Once again, gradual steady 
growth is apparent for Panel B students in mathematics. Within groups, the students’ 
achievement levels varied, while the rates of growth remained consistent. While casual 
inspection of the estimated mean growth curves of the two groups might suggest that they appear 
similar, the rates of growth illustrated are significantly different. The average rate of growth for 
the project group was 31.56 scale score points per year, while that of the control group was 
28.19. Of the four sets of HLM analyses, this was the only finding of differences between 
groups that was statistically significant. 
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Table 12 

Three Level HLM Analysis for Panel B in Mathematics 
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FIXED EFFECT 


Coefficient 


Standard 

Error 


t -ratio 


p value 


Mean status at year 3, yooo 


629.24 


3.02 


208.49 


.000 


Mean growth rate, y 100 


29.87 


1.72 


17.34 


.000 


RANDOM EFFECT 


Variance 

Component 


df 


x 1 


p value 


Level 1 










Temporal variation, e tij 


446.07 








Level 2 (students within groups) 










Individual status at year 3, r 0iJ 


1564.99 


231 


1172.67 


.000 


Individual growth rate, r }i j 
. Level 3 (between groups) 


3.50 


231 


230.21 


>.500 


Group mean status, Uooj 


1.52 


1 


0.26 


>.500 


Group mean growth, u IO j 


3.91 


1 


5.33 


0.020 


LEVEL 1 COEFFICIENT 


Percentage of variance between groups 






Status at year 3, 7T 0i j 


.001 








Growth rate, 7T U j 


.528 









Consequently, the results of the HLM analyses in mathematics were not as consistent as 
those in reading. Gradual, steady growth curves were generally estimated for all students. 
However, for Panel A no variation attributable to group membership was identified, while 
between-group variation was identified for Panel B. In Panel B, the project group’s higher rates 
of growth in mathematics were found to be the significant source of variation. Nonetheless, the 
levels of mathematics achievement were not significantly different. 
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Figure 4. Estimated true mathematics scaled scores for 30 randomly selected students in the 
project and control groups of Panel B. The predicted group means are also given, and 
represented by thick, dashed lines. 



Table 13 

Values of Panel B’s Estimated Parameters in Mathematics 



Group 



n 



Group Fixed Value Individual Empirical Bayes Estimates 

Intercept Slope Intercept Slope 

Mean Standard Mean Standard 

Deviation Deviation 



Panel B 



Project 


108 


628.19 


31.56 


-1.06 


36.49 


.02 


.31 


Control 


127 


630.28 


28.19 


.89 


38.93 


-.01 


.34 



Note. The fixed value parameters remain constant within the group. 
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DISCUSSION 

Hierarchical linear modeling (HLM) provides a somewhat different view of data than the 
more traditional alternative, Analysis of Variance (ANOVA). Unlike ANOVA, HLM analysis 
facilitates the examination of both achievement levels and growth over time. However, the 
analysis is based on estimated true scores, which precludes the examination of achievement 
levels at discrete points in time. Figures 5 and 6 provide a comparison of the project students’ 
observed and estimated true scores across the three years of the project, in reading and 
mathematics, respectively. Note that only the project students’ scores are presented for 
illustrative purposes. 




Figure 5. Comparison of the project students' observed and estimated true reading scores 
across the three years of the project. 



The differences between these two sets of scores point out one difference between the 
ANOVA and HLM procedures. The level of precision available through the interpretation of 
group results in a traditional ANOVA is obscured in the HLM analysis. Yearly fluctuations in 
the groups’ performance, as indicated by mean observed scores are apparent and traceable in 
ANOVA. Trends presented in the HLM analysis smooth the effect of such yearly fluctuations. 
For example, Figure 6 shows that the mathematics scores of students in Panel A showed a 
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marked improvement in year 2, after initial low scores in year 1, but a return to comparatively 
low scores in year 3. The estimated true scores of this group show consistent, gradual growth. 




Figure 6. Comparison of the project students' observed and estimated true mathematics 
scores across the three years of the project. 



Despite the loss of precision available in the interpretation of observed scores in the 
ANOVA procedure, the HLM procedure offers distinct advantages to practitioners in program 
evaluation. First, although the data were collected at uniform time periods for all subjects in the 
current research, this is not a requirement of the HLM procedure. Data may be collected at 
varying time points. As such, the estimated growth curves obtained from HLM analyses are 
intended to be meaningful along the length of the curve, and to extend beyond current 
observations. As result, the trajectories may be used to predict levels of achievement into the 
future. 

Secondly, traditional ANOVA techniques do not allow for the examination of growth 
curves for individual students. Potentially, growth curves can be examined in more detail to 
discover which students might benefit most from a particular program or intervention. Finally, 
the ability of the HLM procedure to examine embedded structures in the data renders it more 
sensitive, and it is thus better able to distinguish the sources of variation in a set of data. This 
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ability would facilitate decision making regarding the adoption and continuation of programs in 
an educational setting. 

The Edison Project Students’ Achievement 

The primary question that guided this study was: Did the students who attended The 
Edison Project’s (currently Edison Schools Inc.) school make greater academic progress than 
comparable students who attended other schools in the district? This question can now be 
addressed. It has been shown that the project students’ performance in reading and mathematics 
prior to enrollment in the project school was comparable to that of the students’ in the control 
group. The results of the analyses conducted after the project students had received instruction 
by means of the Edison model for three years yielded some surprising results. 

The results of the HLM analyses indicated that the students’ scores in reading increased 
over the three years of the study, as would be expected. However, the analyses for both Panels A 
and B failed to identify any statistically significant differences in reading achievement between 
the project students and their counterparts in the control group. Thus, the project students’ 
performance in reading was only comparable to that of the control group. 

The results of the analyses in mathematics were not as consistent. Again, significant 
levels of growth across the three year period were found. However, for Panel A no variation 
attributable to group membership was identified, while the project group’s higher rates of growth 
in mathematics were a significant source of variation in Panel B. Nonetheless, the levels of 
mathematics achievement at the end of the three-year period were not significantly different in 
either group. 

Thus, at the end of the three-year study, the reading and mathematics achievement of 
both the Panel A and B students at The Edison Project’s school was still only comparable to that 
of the students attending other schools in Miami-Dade County. The educational program of the 
Edison model was not superior to the standard program offered at other schools in the district. 

Limitations in the Research 

One limitation of this study was that only three time-ordered observations were available 
for inclusion in the HLM analyses. More than three observations would be required to project 
curvilinear growth models, which are often appropriate when assessing growth in achievement 
over time. More observations conducted over a longer period of time would provide more 
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reliable estimates of true patterns of growth for the project students. In addition, the student 
population in this school consists of a relatively homogeneous group of black students from a 
low socioeconomic neighborhood. A more heterogeneous sample would facilitate the inclusion 
of other predictive variables in the model, and strengthen the external validity of the findings. 
Finally, one should not generalize the results on tests of reading and mathematics to other 
content areas, such as social studies. 

Implications for Future Research 

This study has been limited to assessing the impact of the Edison model on the academic 
performance of students attending one public elementary school in Miami-Dade County, Florida. 
In the future, efforts to replicate the findings of this study should expand the scope by targeting 
schools at multiple levels (i.e., elementary, middle, senior high) in other areas of the country, 
with different demographic characteristics. 

The application of HLM methodology in this study was useful in examining the impact of 
an instructional program on student achievement. However, it is only one means of applying the 
methodology in the evaluation of educational programs. An additional application of the 
methodology is to test variation within and across embedded structures: teachers, classrooms, 
schools, and school districts. Given an adequate sample of schools run by The Edison Project, 
one could determine the sources of variation in student performance, parent satisfaction, teacher 
effects, and many other outcomes. From a theoretical point of view, it may be possible to 
determine how variation in school-level variables affects implementation, and how child-level 
variables relate to individual differences in performance. Thus, HLM methodology appears to be 
a potentially useful tool in the field of program evaluation. 
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